Electronic device training image recognition model and operation method for same

ABSTRACT

A method of training an image recognition model includes: generating a virtual dynamic vision sensor (DVS) image using a virtual simulator; generating label information including information about a correct answer to a result of recognition of the DVS image by the image recognition model, with respect to the DVS image; and training the image recognition model by modifying the image recognition model so that a difference between the result of recognition of the DVS image by the image recognition model and the label information is reduced.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/KR2020/004610 designating the United States, filed on Apr. 6, 2020,in the Korean Intellectual Property Receiving Office and claimingpriority to Korean Patent Application No. 10-2019-0113023, filed on Sep.11, 2019, in the Korean Intellectual Property Office, the disclosures ofwhich are incorporated by reference herein in their entireties.

BACKGROUND Field

The disclosure relates to an electronic device for training an imagerecognition model that recognizes an image, and an operating methodthereof.

Description of Related Art

A dynamic vision sensor (DVS) image captured by a DVS may include pixeldata indicating an amount of change in light sensed by a moving object,unlike existing images including image data with respect to the entireregion of the image. Therefore, unlike existing images, a DVS image hasthe advantage of a fast processing speed because an amount of data to beprocessed is small, and may be utilized in various operations that maybe performed by sensing a moving object.

An image recognition model is an artificial intelligence model forrecognizing a DVS image, and may be used, for example, to recognize amoving object captured from a DVS image, and to extract data related tothe moving object. The image recognition model may be trained based ondata related to a plurality of DVS images.

However, when an amount of data for training the image recognition modelis insufficient, the accuracy and performance of the image recognitionmodel may deteriorate. Therefore, there is a need for a method ofconstructing an image recognition model, having high accuracy andperformance even when the amount of data related to the DVS image fortraining the image recognition model is insufficient.

SUMMARY

Embodiments of the disclosure provide an electronic device for trainingan image recognition model and an operating method thereof.

Embodiments of the disclosure provide a computer-readable recordingmedium having recorded thereon a program for executing the method on acomputer. The disclosure is not limited to the above aspects, and theremay be other aspects of the disclosure.

According to an example embodiments of the disclosure, there is provideda method of training an image recognition model including: generating avirtual dynamic vision sensor (DVS) image using a virtual simulator;generating label information including information about a correctanswer to a result of recognizing the DVS image by the image recognitionmodel with respect to the DVS image; and training the image recognitionmodel by modifying the image recognition model so that a differencebetween the result of recognizing the DVS image by the image recognitionmodel and the label information is minimized.

According to an example embodiment of the disclosure, there is providedan electronic device configured to train an image recognition modelincluding: a memory storing the image recognition model; and at leastone processor configured to: generate a virtual dynamic vision sensor(DVS) image using a virtual simulator, generate label informationcomprising information about a correct answer to a result of recognizingthe DVS image by the image recognition model with respect to the DVSimage, and train the image recognition model by modifying the imagerecognition model so that a difference between the result of recognizingthe DVS image by the image recognition model and the label informationis minimized.

According an example embodiment of the disclosure, there is provided arecording medium having stored therein a program for performing themethod according to the above method.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing detailed description, taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block diagram illustrating an example of training an imagerecognition model according to various embodiments;

FIG. 2 is a block diagram illustrating an example configuration of anelectronic device according to various embodiments;

FIG. 3 is a block diagram illustrating an example configuration of anelectronic device according to various embodiments;

FIG. 4 is a flowchart illustrating an example method of training animage recognition model according to various embodiments;

FIG. 5 is a diagram illustrating an example of a virtual environmentgenerated by a virtual simulator according to various embodiments;

FIG. 6 is a flowchart illustrating an example method of training animage recognition model based on virtual data according to variousembodiments; and

FIG. 7 is a block diagram illustrating an example of an electronicdevice and an external device according to various embodiments.

DETAILED DESCRIPTION

Hereinafter, various example embodiments of the disclosure will bedescribed in greater detail with reference to the accompanying drawings.However, it should be understood that the disclosure may be embodied indifferent ways and is not limited to embodiments described herein. Inaddition, portions irrelevant to the description may be omitted from thedrawings for clarity, and like components are denoted by like referencenumerals throughout the disclosure.

Throughout the disclosure, when an element is referred to as being“connected to” another element, the element may be “directly connectedto” the other element, or the element may also be “electricallyconnected to” the other element with an intervening elementtherebetween. In addition, when an element is referred to as “including”or “comprising” another element, unless otherwise stated, the elementmay further include or comprise yet another element rather than precludethe yet other element.

Functions related to artificial intelligence according to the disclosuremay be operated through a processor and a memory. The processor mayinclude at least one processor. In this regard, the at least oneprocessor may be a general-purpose processor such as a centralprocessing unit (CPU), an application processor (AP), or a digitalsignal processor (DSP), a dedicated graphics processor such as agraphics processing unit (GPU) or a vision processing unit (VPU), or anartificial intelligence-dedicated processor such as a neural processingunit (NPU). The at least one processor may be controlled to processinput data according to a predefined operation rule stored in the memoryor an artificial intelligence model. Alternatively, when the at leastone processor is an artificial intelligence-dedicated processor, theartificial intelligence-dedicated processor may be designed in ahardware structure specialized for processing a specific artificialintelligence model.

The predefined operation rule or the artificial intelligence model aremade through training. Here, the expression “made through training” mayrefer, for example, to an existing artificial intelligence model beingtrained based on a learning algorithm using a large number of pieces oftraining data and thus made into a predefined operation rule or anartificial intelligence model, which is set to fulfill an intendedfeature (or purpose). The training may be performed by a device itself,in which artificial intelligence according to the disclosure isperformed, or may be performed through a separate server and/or system.Examples of the learning algorithm include supervised learning,unsupervised learning, semi-supervised learning, and reinforcementlearning, but are not limited thereto.

An artificial intelligence model may include a plurality of neuralnetwork layers. Each of the neural network layers has a plurality ofweight values and performs a neural network operation through anoperation between an operation result of a previous layer and theplurality of weight values. The plurality of weight values that theneural network layers have may be optimized by a result of training ofthe artificial intelligence model. For example, the plurality of weightvalues may be refined to minimize a loss value or cost value obtained bythe artificial intelligence model during a training process. Anartificial neural network may include a deep neural network (DNN), andmay be, for example, a convolutional neural network (CNN), a deep neuralnetwork (DNN), a recurrent neural network (RNN), a restricted Boltzmannmachine (RBM), a deep belief network (DBN), a bidirectional recurrentdeep neural network (BRDNN), or a deep Q-network, but is not limitedthereto.

Hereinafter, the disclosure will be described in greater detail belowwith reference to the accompanying drawings.

FIG. 1 is a block diagram illustrating an example of training an imagerecognition model 130 according to various embodiments.

Referring to FIG. 1, an electronic device 1000 for training the imagerecognition model 130 according to an embodiment of the disclosure mayobtain a virtual dynamic vision sensor (DVS) image and label information120 with respect to the virtual DVS image through a virtual simulator110, and train the image recognition model 130 based on the obtainedvirtual DVS image and label information 120. The electronic device 1000according to an embodiment of the disclosure may train the imagerecognition model 130 by modifying the image recognition model 130 sothat a difference between label information and a result of recognizingthe virtual DVS image by the image recognition model 130 is minimized.

A method of training the image recognition model 130 according to anembodiment of the disclosure may be performed on various types of imagesas well as a DVS image. For example, the image recognition model 130 maybe trained based on various types of images and label informationgenerated through the virtual simulator 110.

The electronic device 1000 according to an embodiment of the disclosuremay recognize at least one DVS image using the image recognition model130 trained according to an embodiment of the disclosure. For example,as a result of recognizing the DVS image using the image recognitionmodel 130, the electronic device 1000 may obtain information (e.g.,size, location, identification information, movement direction, movementspeed, state, etc. of an object included in the DVS image) about the DVSimage, and perform various operations based on the obtained information.

The electronic device 1000 according to an embodiment of the disclosuremay be implemented in various forms. For example, the electronic device1000 described herein may include a digital camera, a smart phone, alaptop computer, a tablet PC, an electronic book terminal, a digitalbroadcasting terminal, a personal digital assistant (PDA), a portablemultimedia player (PMP), a navigation system, an MP3 player, etc., butis not limited thereto.

The electronic device 1000 described herein may include a wearabledevice that may be worn by a user. The wearable device may include, butis not limited to, at least one of an accessory type device (forexample, a watch, a ring, a wristband, an ankle band, a necklace,glasses, or contact lenses), a head-mounted device (HMD), a fabric orclothing-integrated device (for example, electronic clothes), abody-attached device (for example, a skin pad), or a bio-implantabledevice (for example, an implantable circuit).

The DVS image according to an embodiment of the disclosure may be animage generated by a DVS sensor. The DVS sensor according to anembodiment is an image sensor employing a method in which an iris of aperson receives information, and is a sensor capable of obtaining imagedata of a moving object. For example, the DVS sensor may transmit theimage data to a processor only when there is a local change due to amovement in a pixel unit. The DVS sensor according to an embodiment ofthe disclosure may transmit the image data to the processor only when amoving event occurs. Therefore, the DVS sensor according to anembodiment of the disclosure may not process data when the object is notmoving but measures the moving object when the object is moving andtransmits the data to the processor, thereby preventing and/or reducingwastage of data due to frames continuously transmitted by general imagesensors to the image processor.

The DVS sensor according to an embodiment of the disclosure may addressa problem that a general visual recognition system is vulnerable to afast movement. In addition, the DVS sensor may overcome a blurphenomenon because the DVS sensor receives data on a per-pixel basisrather than on a frame-unit basis.

The DVS sensor according to an embodiment of the disclosure may have aresolution of microseconds. The DVS sensor may have a better temporalresolution than an ultra-high-speed camera that shoots thousands offrames per second (e.g., ultra-high-speed frames >1 K FPS). In addition,according to the DVS sensor, because power consumption and data storagerequirements are also greatly reduced, a dynamic range (a range ofbrightness that s sensor is capable of distinguish) may also remarkablyincrease.

According to an embodiment of the disclosure, because an outline of themoving object is expressed in the image obtained by the DVS sensor, itmay be advantageous to protect the privacy of a monitored object. Inaddition, the DVS sensor may generate the DVS image by sensing themovement of an object even in a dark place with little light.

An image recognition model for recognizing the DVS image according to anembodiment of the disclosure may be trained based on the DVS imagegenerated by the DVS sensor and label information about the DVS image.

The label information according to an embodiment of the disclosure mayinclude information about a correct answer for a result of recognizingthe DVS image by the image recognition model. The label informationaccording to an embodiment of the disclosure may include a plurality ofpieces of information different from each other according to a type ofinformation to be recognized from the DVS image. For example, the labelinformation is information that may be recognized with respect to thesame DVS image, and may include different types of information, such asa state of an object and a moving direction of the object.

According to an embodiment of the disclosure, the image recognitionmodel may be trained based on a pair of at least one DVS image and labelinformation corresponding to each DVS image. For example, the imagerecognition model may be trained by modifying the image recognitionmodel so that a difference between the result of recognizing the DVSimage by the image recognition model and the label information withrespect to the DVS image is minimized.

For example, when the label information includes information indicating“a state in which a person is walking”, the image recognition model maybe trained to output the same or similar result as “a state in which aperson is walking” as the result of recognizing the DVS image by theimage recognition model.

However, when the pair of DVS image and label information for trainingthe image recognition model is not sufficient for the image recognitionmodel to be trained, there is a problem that the performance of theimage recognition model may deteriorate.

For example, because label information needs to be directly input by aperson, when the number of DVS images in which label information ispresent is absolutely small, the image recognition model may not besufficiently trained, which may deteriorate the accuracy and performanceof the image recognition model.

However, according to an embodiment of the disclosure, even when thepair of DVS image and label information for training the imagerecognition model is small, the electronic device 1000 may use thevirtual simulator 110 to sufficiently generate the pair of virtual DVSimage and label information, and thus the image recognition model may besufficiently trained.

In addition, because the outline of a moving object is expressed in theDVS image according to an embodiment of the disclosure, a virtuallygenerated DVS image and a real DVS image captured by the DVS sensor maybe substantially similar in appearance. Therefore, according to theimage recognition model trained based on the pair of DVS image and labelinformation generated using the virtual simulator 110 according to anembodiment of the disclosure, image recognition may be performed even onthe real DVS image with high performance and accuracy.

The virtual simulator 110 according to an embodiment of the disclosuremay generate various types of virtual environments in which DVS imagesmay be captured. For example, the virtual simulator 110 may generatevarious virtual environments such as a house, an office, and a road.

The virtual simulator 110 according to an embodiment of the disclosuremay generate various types of virtual objects that may exist in avirtual environment, and may place the generated virtual objects in thevirtual environment. According to an embodiment of the disclosure, in areal environment corresponding to the virtual environment, an objectthat may be captured as the real DVS image may be placed as a virtualobject in the virtual environment. For example, virtual objects such aspeople, furniture, home appliances, and pets may be placed in a virtualenvironment of a house. In addition, virtual objects such as signs,cars, and lanes may be placed in a virtual environment of the road.

Accordingly, the virtual DVS image according to an embodiment of thedisclosure may be generated based on the virtual environment generatedby the virtual simulator 110 and at least one virtual object placed inthe virtual environment. Also, the virtual DVS image according to anembodiment of the disclosure may be obtained as a plurality of imagesequences in which scenes in which a virtual object moves or changesover time are captured.

The virtual simulator 110 according to an embodiment of the disclosuremay generate the virtual DVS image that may be captured by the DVSsensor in the virtual environment. The virtual simulator 110 accordingto an embodiment of the disclosure may determine from least one cameraview point in which an actual DVS sensor may be located in the virtualenvironment, and generate at least one virtual DVS image captured fromeach camera view point. The at least one virtual DVS image may be animage simultaneously captured from each camera view point.

For example, the virtual simulator 110 may generate the virtual DVSimage captured in the virtual environment by generating a vision imagethat may be captured from the at least one camera view point in thevirtual environment and generating the DVS image from the vision image.The DVS image is not limited to the vision image, and may be generatedfrom other types of images. The disclosure is not limited to theabove-described example, the virtual simulator 110 may generate avirtual DVS image captured in a virtual environment through variousmethods.

The label information with respect to the virtual DVS image according toan embodiment of the disclosure may be generated based on informationabout at least one of the virtual environment or the at least onevirtual object placed in the virtual environment. The label informationaccording to an embodiment of the disclosure may be obtained based onpreviously set information about the virtual environment and the atleast one virtual object used to generate the virtual environment andthe at least one virtual object.

According to an embodiment of the disclosure, the virtual simulator 110may arrange a virtual object in the virtual environment so that thevirtual object is placed or moved according to the previously setinformation. The label information according to an embodiment of thedisclosure may be obtained based on information about predefinedcharacteristic information with respect to at least one of the virtualenvironment or the virtual object.

For example, the virtual simulator 110 may arrange a vehicle in thevirtual environment so that the vehicle moves along a previouslydesignated path in the virtual environment. In addition, after aposition and speed of the vehicle moving along the previously designatedpath are also previously set by the virtual simulator 110, the vehiclemay be placed in the virtual environment according to the setinformation. Accordingly, label information with respect to a virtualDVS image in which the vehicle is captured may be obtained based on apath, location, speed, etc., which is information about characteristicinformation of the vehicle, which is previously set by the virtualsimulator 110 to arrange the vehicle.

The label information according to an embodiment of the disclosure maybe obtained based on characteristic information predefined by thevirtual simulator 110 when the virtual environment and the virtualobject are generated. Accordingly, the label information with respect tothe virtual DVS image according to an embodiment of the disclosure mayinclude more accurate information than label information directly inputby the person with respect to a real image.

In addition, the label information according to an embodiment of thedisclosure may be automatically obtained based on characteristicinformation predefined by the virtual simulator 110. The labelinformation according to an embodiment of the disclosure may beautomatically obtained based on the predefined characteristicinformation, when the virtual environment or the virtual object isgenerated by the virtual simulator 110. Accordingly, the labelinformation according to an embodiment of the disclosure may be easilyand quickly obtained by the electronic device 1000.

According to an embodiment of the disclosure, the label information maybe automatically obtained whenever a virtual DVS image is generatedbased on the characteristic information predefined by the virtualsimulator 110. Accordingly, according to an embodiment of thedisclosure, as data for training the image recognition model 130, alarge number of pairs of virtual DVS images and label information may bequickly and accurately generated in a short time.

The image recognition model 130 according to an embodiment of thedisclosure may be trained based on the virtual DVS image generated bythe virtual simulator 110 and the label information with respect to thevirtual DVS image.

The electronic device 1000 according to an embodiment of the disclosuremay train the image recognition model 130 by modifying at least onecomponent included in the image recognition model 130 so that adifference between the label information and the information about theresult of recognizing the DVS image by the image recognition model 130is minimized and/or reduced. For example, a structure of a node, aweight value, a bias value, etc. included in the image recognition model130 may be modified as the image recognition model 130 is trained.

The image recognition model 130 according to an embodiment of thedisclosure may be a data recognition model based on a neural networksuch as a convolutional neural network (CNN), a deep neural network(DNN), a recurrent neural network (RNN), and a multi-layer perceptron(MLP) used to classify and detect an object in an image. The imagerecognition model 130 according to an embodiment of the disclosure isnot limited to the above-described example, and may include varioustypes of artificial intelligence models.

According to an embodiment of the disclosure, based on the imagerecognition model 130, the image recognition result may include, forexample, information about an object recognized from an image,information about a location of the object recognized from the image,and information about a movement of the object recognized from theimage. For example, when an image input to a data recognition model is aDVS image in which a “vehicle” is captured, a recognition result of theimage of the data recognition model may include the “vehicle”.

Accordingly, according to an embodiment of the disclosure, the imagerecognition model 130 may be trained so that a result close to the“vehicle” may be output when a real DVS image similar to a virtual DVSimage is input to the image recognition model 130 based on the virtualDVS image including the “vehicle” and label information.

According to an embodiment of the disclosure, the image recognitionmodel 130 may be trained based on a large number of pairs of virtual DVSimages and label information generated quickly and accurately by thevirtual simulator 110. Accordingly, even when the real DVS image andlabel information for training the image recognition model 130 areinsufficient, the electronic device 1000 according to an embodiment ofthe disclosure may quickly obtain the image recognition model 130 havinghigh accuracy and performance, based on pairs of virtual DVS images andlabel information.

The electronic device 1000 according to an embodiment of the disclosuremay use an external server (not shown) to train the image recognitionmodel 130 for recognizing a DVS image.

The external server according to an embodiment of the disclosure may beimplemented as at least one computer device. The external server may bedistributed in the form of a cloud and may provide commands, codes,files, contents, etc.

The external server may perform operations that the electronic device1000 may execute. For example, the external server may generate thevirtual DVS image and the label information for training the imagerecognition model 130 according to a request of the electronic device1000. The external server may train the image recognition model 130based on the generated virtual DVS image and label information, andtransmit the trained image recognition model 130 to the electronicdevice 1000. The external server may transmit, to the electronic device1000, a result of recognizing a real DVS image by the trained imagerecognition model 130 according to an embodiment of the disclosure.

According to an embodiment of the disclosure, the image recognitionmodel 130 may be trained on the electronic device 1000 and a result ofrecognizing an image by the image recognition model 130 may be outputwithout data transmission and reception with an external server (notshown), according to an on-device AI technology. For example, theelectronic device 1000 may train the image recognition model 130according to an embodiment of the disclosure according to the generatedDVS image, based on various types of information collected by theelectronic device 1000 in real time, without having to use big datastored in the external server.

According to the on-device AI technology, the electronic device 1000 maybe trained by itself based on data collected by itself, and may make adecision by itself based on a trained AI model. According to theon-device AI technology, the electronic device 1000 operates by itselfwithout transmitting the collected data to the outside, and thus thereis an advantage in terms of protection of personal information of a userand a data processing speed.

For example, the electronic device 1000 may operate using the on-deviceAI technology without a connection with the external server, accordingto whether a network environment of the electronic device 1000 isunstable, or whether only the information collected by the electronicdevice 1000 is sufficient to perform an operation according to anembodiment of the disclosure according to the AI model trained by theelectronic device 1000 without having to use big data.

However, the electronic device 1000 is not limited to operatingaccording to the on-device AI technology, and may perform the operationaccording to an embodiment of the disclosure through data transmissionand reception with an external server or an external device. Theelectronic device 1000 may perform the operation according to anembodiment of the disclosure by combining the above-described on-deviceAI technology and a method through data transmission and reception withthe external server.

For example, the operation according to an embodiment of the disclosuremay be performed according to the method through the external serverwhen an operation through the external server is more advantageous interms of the data processing speed according to a network environmentand a computing power of the electronic device 1000, or when the methodthrough the external server is more advantageous than the on-device AItechnology such as data that does not include the personal informationof the user is transmitted to the external server.

FIG. 2 is a block diagram illustrating an example configuration of theelectronic device 1000 according to various embodiments.

FIG. 3 is a block diagram illustrating an example configuration of theelectronic device 1000 according to various embodiments.

Referring to FIG. 2, the electronic device 1000 may include a processor(e.g., including processing circuitry) 1300 and a memory 1700. However,not all of the components shown in FIG. 2 are indispensable componentsof the electronic device 1000. The electronic device 1000 of FIG. 2 maybe implemented by more components or less components than the componentsshown in FIG. 2.

For example, as shown in FIG. 3, the electronic device 1000 according toan embodiment of the disclosure may further include a communicator(e.g., including communication circuitry) 1520, an outputter (e.g.,including output circuitry) 1020, a user inputter (e.g., including inputcircuitry) 1100, a sensing unit (e.g., including at least one sensor)1400, and an audio/video (A/V) inputter (e.g., including A/V inputcircuitry) 1600, in addition to the processor 1300 and the memory 1700.

The user inputter 1100 may include various circuitry for inputting datafor a user to control the electronic device 1000. For example, the userinputter 1100 may include a keypad, a dome switch, a touch pad (a touchcapacitive type, a pressure resistive type, an infrared beam sensingtype, a surface acoustic wave type, an integral strain gauge type, apiezoelectric type, etc.), a jog wheel, a jog switch, etc. but is notlimited thereto.

According to an embodiment of the disclosure, the user inputter 1100 mayreceive a user input for training the image recognition model 130.

The outputter 1200 may include various output circuitry and output anaudio signal, a video signal, or a vibration signal, and the outputter1200 may include a display 1210, a sound outputter 1220, and a vibrationmotor 1230.

The display 1210 displays and outputs information processed by theelectronic device 1000. According to an embodiment of the disclosure,the display 1210 may display a virtual DVS image generated by thevirtual simulator 110. Also, the display 1210 according to an embodimentof the disclosure may display a result of recognizing the DVS image bythe image recognition model 130.

When the display 1210 and the touch pad form a layer structure toprovide a touch screen, the display 1210 may be used as an input devicein addition to an output device. The display 1210 may include at leastone of a liquid crystal display, a thin film transistor-liquid crystaldisplay, an organic light-emitting diode, a flexible display, athree-dimensional (3D) display, or an electrophoretic display. Also,according to an implementation form of the electronic device 1000, theelectronic device 1000 may include two or more displays 1210.

The sound outputter 1220 may include various circuitry that outputsaudio data received from the communicator 1500 or stored in the memory1700.

The vibration motor 1230 may output a vibration signal. Also, thevibration motor 1230 may output a vibration signal when a touch is inputto the touch screen.

The sound outputter 1220 and the vibration motor 1230 according to anembodiment of the disclosure may output information related to a resultof training the image recognition model 130 based on the virtual DVSimage and label information, or the result of recognizing the DVS imageby the image recognition model 130.

The processor 1300 may include various processing circuitry andgenerally controls the overall operation of the electronic device 1000.For example, the processor 1300 may generally control the user inputter1100, the outputter 1200, the sensing unit 1400, the communicator 1500,and the A/V inputter 1600 by executing programs stored in the memory1700.

The electronic device 1000 may include at least one processor 1300. Forexample, the electronic device 1000 may include various types ofprocessor such as a central processing unit (CPU), a graphics processingunit (GPU), and a neural processing unit (NPU).

The processor 1300 may be configured to process commands of a computerprogram by performing basic arithmetic, logic, and input/outputoperations. The commands may be provided to the processor 1300 from thememory 1700 or may be received through the communicator 1500 andprovided to the processor 1300. For example, the processor 1300 may beconfigured to execute the commands according to program codes stored ina recording device such as memory.

The processor 1300 according to an embodiment of the disclosure maygenerate a virtual DVS image using the virtual simulator 110 and maygenerate label information with respect to the DVS image. The labelinformation according to an embodiment of the disclosure may includeinformation about a correct answer to the result of recognizing the DVSimage by the image recognition model 130. The label informationaccording to an embodiment of the disclosure may be obtained wheneverthe DVS image is generated, based on predefined characteristicinformation about at least one of a virtual environment or a virtualobject placed in the virtual environment, which is previously set by thevirtual simulator 110 when the DVS image is generated.

The processor 1300 according to an embodiment of the disclosure maytrain the image recognition model 130 by modifying at least onecomponent included in the image recognition model 130 so that adifference between the label information and the information about theresult of recognizing the DVS image by the image recognition model 130is minimized.

In addition, the processor 1300 according to an embodiment of thedisclosure may determine at least one camera view point in the virtualenvironment generated by the virtual simulator 110, and generate atleast one virtual DVS image simultaneously captured from each cameraview point. Accordingly, the image recognition model according to anembodiment of the disclosure may be trained based on the at least onevirtual DVS image with respect to the at least one camera view point.

In addition, the processor 1300 according to an embodiment of thedisclosure may generate a virtual environment based on information abouta surrounding environment in which a real DVS image that may berecognized by the image recognition model may be captured, and, based onthe virtual environment, generate a virtual DVS image. The virtual DVSimage according to an embodiment of the disclosure may be generatedbased on changed information about the surrounding environment wheneverthe information about the surrounding environment changes by more than areference value.

The processor 1300 according to an embodiment of the disclosure mayrecognize a DVS image using the image recognition model 130 and output aresult thereof.

The sensing unit 1400 may include various sensors and sense a state ofthe electronic device 1000 or a state around the electronic device 1000,and may transfer sensed information to the processor 1300.

The sensing unit 1400 may include at least one of a geomagnetic sensor1410, an acceleration sensor 1420, a temperature/humidity sensor 1430,an infrared sensor 1440, a gyroscope sensor 1450, and a position sensor.(e.g., GPS) 1460, a barometric pressure sensor 1470, a proximity sensor1480, or an RGB sensor (illuminance sensor) 1490, but is not limitedthereto.

The sensing unit 1400 according to an embodiment of the disclosure mayfurther include a DVS sensor for capturing a DVS image.

According to an embodiment of the disclosure, based on the informationsensed by the sensing unit 1400, a virtual environment may be generatedor an object of the virtual environment may be generated and placed. Theelectronic device 1000 according to an embodiment of the disclosure maygenerate a virtual environment and an object that are highly likely tobe captured as real DVS images by the DVS sensor of the electronicdevice 1000, based on the information sensed by the sensing unit 1400.For example, based on the information of the electronic device 1000, theelectronic device 1000 may predict an environment in which theelectronic device 1000 is currently placed, and based on the predictedenvironment, generate the virtual environment and the object of thevirtual environment that are highly likely to be captured as real DVSimages

Accordingly, according to an embodiment of the disclosure, the imagerecognition model 130 may be trained based on a virtual DVS image thatis highly likely to be captured as a real DVS image.

The communicator 1500 may include one or more components, each includingvarious communication circuitry, that allow the electronic device 1000to communicate with the server 2000 or an external device (not shown).For example, the communicator 1500 may include a short-rangecommunicator 1510, a mobile communicator 1520, and a broadcast receiver1530.

The short-range wireless communicator 1510 may include a Bluetoothcommunicator, a Bluetooth Low Energy (BLE) communicator, a near fieldcommunicator, a wireless local area network (WLAN) (Wi-Fi) communicator,a Zigbee communicator, an Infrared Data Association (IrDA) communicator(not shown), a Wi-Fi Direct (WFD) communicator, an ultra wideband (UWB),and an Ant+ communicator, but is not limited thereto.

The mobile communicator 1520 transmits and receives wireless signals toand from at least one of a base station, an external terminal, or aserver on a mobile communication network. Here, the wireless signal mayinclude various types of data according to transmission/reception of avoice call signal, a video call signal, or a text/multimedia message.

The broadcast receiver 1530 receives a broadcast signal and/orbroadcast-related information from the outside through a broadcastchannel. The broadcast channel may include a satellite channel and aterrestrial channel. According to an embodiment of the disclosure, theelectronic device 1000 may not include the broadcast receiver 1530.

According to an embodiment of the disclosure, the communicator 1500 maytransmit and receive data required for training the image recognitionmodel 130.

The A/V inputter 1600 may include various components including variouscircuitry for inputting an audio signal or a video signal, and mayinclude a camera 1610, a microphone 1620, etc. The camera 1610 mayobtain an image frame such as a still image or a moving image through animage sensor in a video call mode or a photographing mode. The imagecaptured through the image sensor may be processed through the processor1300 or a separate image processing unit (not shown).

The microphone 1620 receives an external sound signal and processes thesound signal as electrical speech data.

The memory 1700 may store a program for processing and controlling theprocessor 1300, and may also store data input to or output from theelectronic device 1000.

The memory 1700 according to an embodiment of the disclosure may storeinformation necessary for generating a virtual DVS image and labelinformation and training the image recognition model 130. For example,the memory 1700 may store the image recognition model 130. Also, thememory 1700 according to an embodiment of the disclosure may storeinformation about the virtual simulator 110 that generates the virtualDVS image.

The memory 1700 may include at least one of a flash memory type storagemedium, a hard disk type storage medium, a multimedia card micro typestorage medium, card type memory (e.g., secure digital (SD) memory,eXtreme Digital (XD) memory, etc.), random access memory (RAM), staticrandom access memory (SRAM), read-only memory (ROM), electricallyerasable programmable read-only memory (EEPROM), programmable read-onlymemory (PROM), magnetic memory, a magnetic disk, or an optical disk.

The programs stored in the memory 1700 may be classified into aplurality of modules, for example, a UI module 1710, a touch screenmodule 1720, a notification module 1730, etc. according to functionsthereof.

The UI module 1710 may provide a specialized UI, a graphic userinterface (GUI), etc. that interact with the electronic device 1000 foreach application. The touch screen module 1720 may sense a touch gestureof the user on a touch screen and may transfer information about thetouch gesture to the processor 1300. The touch screen module 1720according to some embodiments may recognize and analyze a touch code.The touch screen module 1720 may be configured as separate hardwareincluding a controller.

Various sensors may be provided inside or near the touch screen to sensea touch or a proximity touch of the touch screen. An example of a sensorfor sensing a touch with respect to the touch screen includes a tactilesensor. The tactile sensor refers to a sensor that senses a touch of aspecific object to the extent or higher than that felt by a human. Thetactile sensor may sense various information such as roughness of acontact surface, hardness of a contact object, and a temperature of acontact point.

The touch gesture of the user may include tap, touch and hold, doubletap, drag, panning, flick, drag and drop, swipe, etc.

The notification module 1730 may generate a signal for notifying theoccurrence of an event in the electronic device 1000.

FIG. 4 is a flowchart illustrating an example method of training theimage recognition model 130 according to various embodiments.

Referring to FIG. 4, in operation 410, the electronic device 1000according to an embodiment of the disclosure may generate a virtual DVSimage using the virtual simulator 110. The electronic device 1000according to an embodiment of the disclosure may generate a virtualenvironment through the virtual simulator 110 and may generate thevirtual DVS image based on the virtual environment.

The virtual simulator 110 according to an embodiment of the disclosuremay generate the virtual environment based on information about asurrounding environment in which a real DVS image recognized by theimage recognition model may be captured. For example, based oninformation sensed by various sensors provided in the electronic device1000, the electronic device 1000 may obtain information about thesurrounding environment, and transmit the information about thesurrounding environment to the virtual simulator 110.

The information about the surrounding environment according to anembodiment of the disclosure may include information about thesurrounding environment of the electronic device 1000 including a DVSsensor that captures a DVS image, or a device other than the electronicdevice 1000.

Accordingly, according to an embodiment of the disclosure, the virtualsimulator 110 may generate the virtual environment based on theinformation about the surrounding environment of the electronic device1000 and may generate the virtual DVS image based on the generatedvirtual environment.

The information about the surrounding environment according to anembodiment of the disclosure may include a variety of information thatmay be sensed by a sensor of the electronic device 1000, such as alocation and a movement state of the electronic device 1000, informationsensed with respect to an object around the electronic device 1000, etc.

The electronic device 1000 according to an embodiment of the disclosuremay determine whether the information about the surrounding environmentobtained by the electronic device 1000 has changed by more than areference value. The electronic device 1000 according to an embodimentof the disclosure may transmit the changed information about thesurrounding environment to the virtual simulator 110 whenever theinformation about the surrounding environment changes by more than thereference value.

The virtual simulator 110 according to an embodiment of the disclosuremay generate the virtual environment whenever the changed informationabout the surrounding environment is obtained from the electronic device1000, and generate the virtual DVS image based on the generated virtualenvironment. Accordingly, according to an embodiment of the disclosure,whenever the information about the surrounding environment changes bymore than the reference value, the virtual DVS image may be generatedbased on the changed information about the surrounding environment.Also, according to an embodiment of the disclosure, the imagerecognition model may be continuously trained based on the changedinformation about the surrounding environment of the electronic device1000.

In operation 420, the electronic device 1000 according to an embodimentof the disclosure may generate label information with respect to thevirtual DVS image. The label information according to an embodiment ofthe disclosure may include information about at least one correct answerto a result of recognizing the virtual DVS image by the imagerecognition model. The image recognition model according to anembodiment of the disclosure may be modified so that a differencebetween the result of the image recognition model and the correct answerinformation is minimized, so that the image recognition model may beupdated.

The label information according to an embodiment of the disclosure maybe determined based on information about the virtual environmentgenerated by the virtual simulator 110. According to an embodiment ofthe disclosure, when a virtual object is placed in the virtualenvironment after the virtual environment is generated, the informationabout the virtual environment according to an embodiment of thedisclosure may be determined according to predefined characteristicinformation about the virtual environment and the virtual object.

For example, when the virtual object is placed to move at a constantspeed in the virtual environment, the label information may bedetermined based on identification information and movement informationof the virtual object.

Because the label information according to an embodiment of thedisclosure is generated based on the virtual environment and the virtualobject implemented according to information previously set by thevirtual simulator 110, the label information may be automaticallygenerated whenever the virtual DVS image is generated. In addition,because the label information may be determined according to informationpreviously set by the virtual simulator 110, the label information mayinclude accurate and detailed information.

In operation 430, the electronic device 1000 according to an embodimentof the disclosure may train the image recognition model for recognizingthe real DVS image based on the virtual DVS image and the labelinformation corresponding to the virtual DVS image.

According to an embodiment of the disclosure, even when the DVS imagedata for training the image recognition model is insufficient, the imagerecognition model may be sufficiently trained based on the virtual DVSimage generated by the virtual simulator 110. In addition, because thelabel information with respect to the virtual DVS image according to anembodiment of the disclosure may be determined according to accurate anddetailed information previously set by the virtual simulator 110, theimage recognition model is trained based on the label information, andthus the performance of the image recognition model may be better.

FIG. 5 is a diagram illustrating an example of a virtual environmentgenerated by the virtual simulator 110 according to various embodiments.

Referring to FIG. 5, the virtual simulator 110 according to anembodiment of the disclosure may generate a virtual environment 500 inwhich, for example, an environment inside a house is implemented, andplace a plurality of objects 511, 512, and 520 in the virtualenvironment 500.

In an embodiment of the disclosure, the plurality of objects 511, 512,and 520 that may be placed in the virtual environment 500 may be objectsthat are likely to be captured as a real DVS image in a real environmentcorresponding to the virtual environment 500.

In the virtual environment 500 according to an embodiment of thedisclosure, a sofa 512 and a table 511, which are fixed objects, and arobot cleaner 520, which is an object that captures the virtual DVSimage while moving, may be placed. The sofa 512, the table 511, and therobot cleaner 520 may be placed in previously designated positions inthe virtual environment 500. In addition, the robot cleaner 520 may beplaced to move in the virtual environment 500 according to previouslydesignated path and speed.

The virtual DVS image according to an embodiment of the disclosure maybe generated at each camera view point with respect to the plurality ofDVS sensors 521, 522, and 523 placed in the virtual environment 500.According to an embodiment of the disclosure, in the virtual environment500, the image recognition model 330 may be trained based on a pluralityof virtual DVS images simultaneously captured from different camera viewpoints. In addition, the virtual DVS image according to an embodiment ofthe disclosure may be obtained as a plurality of image sequencesincluding a scene in which the robot cleaner 520 moves captured fromdifferent camera view points over time.

The DVS sensors 522 and 523 placed in fixed positions may obtain avirtual DVS image in which the robot cleaner 520 which is a movingobject is sensed, except for the table 511 and sofa 512 which are fixedobjects. The DVS sensor 521 disposed in the moving robot cleaner 520 maysense even a fixed object as a moving object in the DVS image due to themovement of the DVS sensor 521. Accordingly, the DVS sensor 521 disposedin the robot cleaner 520 may obtain the virtual DVS image in which notonly the moving object but also the table 511 and the sofa 512 which arefixed objects are sensed.

Accordingly, according to an embodiment of the disclosure, the imagerecognition model 330 may be trained based on more precise and a lot ofdata, according to a plurality of virtual DVS images captured fromdifferent view points and in different moving states.

FIG. 6 is a flowchart illustrating an example method of training animage recognition model based on virtual data according to variousembodiments.

The virtual data according to an embodiment of the disclosure mayinclude a pair of a virtual DVS image and label information generatedbased on a virtual environment.

In operation 610, the electronic device 1000 according to an embodimentof the disclosure may generate the virtual environment with respect to aDVS image in order to obtain the virtual DVS image. The virtualenvironment according to an embodiment of the disclosure may begenerated by previously defining characteristic information of thevirtual environment (e.g., size, object, brightness, etc. of the virtualenvironment). For example, based on surrounding environment informationsensed by the electronic device 1000, characteristic information relatedto the virtual environment may be previously set.

In operation 620, the electronic device 1000 according to an embodimentof the disclosure may generate the virtual DVS image based on thevirtual environment. After generating the virtual environment, theelectronic device 1000 according to an embodiment of the disclosure maydetermine at least one view point from which the virtual DVS image iscaptured.

The at least one view point from which the virtual DVS image is capturedaccording to an embodiment of the disclosure may be fixed or moved.According to an embodiment of the disclosure, when the at least one viewpoint from which the virtual DVS image is captured is a moving viewpoint, information about movement, such as a movement path and speed,may be set, and the virtual DVS image may be generated according to theset information.

For example, when the virtual DVS image is captured by a DVS sensorincluded in a robot cleaner that is placed in the virtual environmentand moves in a house, a movement path and speed of the robot cleaner maybe set based on characteristic information of the robot cleaner.According to the set movement information, the virtual DVS imagecaptured by the DVS sensor of the robot cleaner may be generated.

In operation 630, the electronic device 1000 according to an embodimentof the disclosure may generate label information with respect to thevirtual DVS image generated in operation 620 based on the virtualenvironment. The label information according to an embodiment of thedisclosure may include correct answer information indicating a correctanswer to a result of recognizing the virtual DVS image by the imagerecognition model.

The label information according to an embodiment of the disclosure maybe obtained when the correct answer information is determined, based onthe above-described previously set characteristic information withrespect to the virtual environment, previously set information about themoving path and speed of the DVS sensor that captures the DVS image, andpreviously set information regarding the virtual object included in theDVS image (e.g., location, movement speed, movement path, identificationinformation, and state), etc.

The label information according to an embodiment of the disclosure maybe determined based on information that needs to be previously set inorder to generate the virtual environment and place the virtual object.Accordingly, at a time point at which the virtual DVS image isgenerated, the information used to determine the label information maybe information already stored in the electronic device 1000 to generatethe virtual environment and place the virtual object. The labelinformation according to an embodiment of the disclosure may beautomatically determined without another input from a user based on theinformation already stored in the electronic device 1000 in relation tothe virtual environment.

In addition, the label information according to an embodiment of thedisclosure may include at least one correct answer informationindicating a correct answer to information that may be output by theimage recognition model. For example, when identification information,state information, movement information, etc. about an object recognizedin an image may be output as an image recognition result by the imagerecognition model, the label information may include at least onecorrect answer information respectively corresponding to the objectrecognition information and object state information, object movementinformation, etc.

In operation 640, the electronic device 1000 according to an embodimentof the disclosure may generate virtual data including the at least onevirtual DVS image and the label information corresponding to each DVSimage. In operation 650, the electronic device 1000 according to anembodiment of the disclosure may train the image recognition model basedon the virtual data.

According to an embodiment of the disclosure, the image recognitionmodel may be trained by modifying the image recognition model so that arecognition result by the image recognition model with respect to thevirtual DVS image has a value close to the correct answer information ofthe label information.

FIG. 7 is a block diagram illustrating an example of the electronicdevice 1000 and an external device 700 according to various embodiments.

Referring to FIG. 7, an image recognition model trained by theelectronic device 1000 according to an embodiment of the disclosure maybe transmitted to the external device 700.

The external device 700 according to an embodiment of the disclosure mayinclude a device including the image recognition model trained by theelectronic device 1000 and recognizing a real DVS image obtained by theexternal device 700 based on the image recognition model. For example,the external device 700 may include a robot cleaner, a smartrefrigerator, a smart TV, a camera, etc. that may be connected to theelectronic device 1000, and is not limited to the above-describedexample, but may include various types of devices.

The electronic device 1000 according to an embodiment of the disclosuremay generate the virtual DVS image that may be captured by each externaldevice 700 and train the image recognition model based on the virtualDVS image. For example, the electronic device 1000 may generate thevirtual DVS image that may be captured by a DVS sensor provided in therobot cleaner according to a movement path and speed of the robotcleaner among the external devices 700. In addition, the electronicdevice 1000 may generate a virtual DVS image that may be captured by aDVS sensor provided in each of the smart TV and the smart refrigeratoraccording to locations of the smart TV and the smart refrigerator amongthe external devices 700.

The electronic device 1000 according to an embodiment of the disclosuremay obtain label information with respect to the virtual DVS image thatmay be captured by each of the external devices 700 based on previouslyset information so as to generate the virtual environment. Thepreviously set information may be characteristic information predefinedwith respect to at least one of a virtual environment or a virtualobject of the virtual environment.

The electronic device 1000 according to an embodiment of the disclosuremay train the image recognition model based on the virtual DVS image andthe label information generated with respect to the at least oneexternal device 700, and transmit the trained image recognition model tothe at least one external device 700.

The at least one external device 700 according to an embodiment of thedisclosure may recognize the DVS image obtained by each external device700 based on the image recognition model received from the electronicdevice 1000.

For example, the smart TV among the external devices 700 may sense amovement of a user from the DVS image captured by the smart TV using theimage recognition model and determine whether the user watches the TV.The smart TV according to an embodiment of the disclosure may performvarious operations, for example, an operation of turning off or on thepower of the smart TV, based on whether the user watches the TV.

As another example, the robot cleaner among the external devices 700 maypredict a movement path of the user from the DVS image captured by therobot cleaner using the image recognition model. The robot cleaneraccording to an embodiment of the disclosure may determine the movementpath of the robot cleaner based on the movement path of the user, andperform floor cleaning while moving according to the determined path.

In addition, the electronic device 1000 according to an embodiment ofthe disclosure may receive the DVS image obtained by each externaldevice 700 rather than transmitting the image recognition model to theat least one external device 700. The electronic device 1000 accordingto an embodiment of the disclosure may recognize the DVS image receivedfrom each external device 700 based on the trained image recognitionmodel according to an embodiment of the disclosure. The electronicdevice 1000 may perform an operation according to a result ofrecognizing the DVS image received from each external device 700 ortransmit information related to the result to each external device 700.

According to an embodiment of the disclosure, even when data fortraining the image recognition model is not sufficient for training theimage recognition model, based on a pair of DVS image and labelinformation generated using a virtual simulator, the image recognitionmodel may be sufficiently trained.

An embodiment of the disclosure may also be realized in a form of arecording medium including instructions executable by a computer, suchas a program module executed by a computer. A computer-readablerecording medium may be an arbitrary available medium accessible by acomputer, and examples thereof include all volatile and non-volatilemedia and separable and non-separable media. Further, examples of thecomputer-readable recording medium may include a computer storage mediumand a communication medium. Examples of the computer storage mediuminclude all volatile and non-volatile media and separable andnon-separable media, which have been implemented by an arbitrary methodor technology, for storing information such as computer-readableinstructions, data structures, program modules, and other data. Thecommunication medium typically includes a computer-readable instruction,a data structure, or a program module, and includes an arbitraryinformation transmission medium.

In addition, the term such as “ . . . unit” or “ . . . portion” usedherein may refer to a hardware component such as a processor or acircuit, and/or a software component executed by the hardware componentsuch as a processor.

It will be understood by one of ordinary skill in the art that theembodiments of the disclosure are provided for illustration and may beimplemented in different ways without departing from the spirit andscope of the disclosure. Therefore, it should be understood that theforegoing example embodiments of the disclosure are provided forillustrative purposes only and are not to be understood in any way aslimiting the disclosure. For example, each component described as asingle type may be implemented in a distributed manner, and likewise,components described as being distributed may be implemented as acombined type.

The scope of the disclosure includes the appended claims and equivalentsthereof, and any changes or modifications derived from the appendedclaims and equivalents thereof should be understood as falling withinthe scope of the disclosure. It should also be understood that any ofthe embodiment(s) described herein may be used in conjunction with anyother embodiment(s) described herein.

What is claimed is:
 1. A method of training an image recognition model,the method comprising: generating a virtual dynamic vision sensor (DVS)image using a virtual simulator; generating label information comprisinginformation about a correct answer to a result of recognition of the DVSimage by the image recognition model with respect to the DVS image; andtraining the image recognition model by modifying the image recognitionmodel so that a difference between the result of recognition of the DVSimage by the image recognition model and the label information isreduced.
 2. The method of claim 1, wherein the virtual DVS image isgenerated based on a virtual environment generated by the virtualsimulator, and a virtual object placed in the virtual environment. 3.The method of claim 2, wherein the label information is obtained basedon predefined characteristic information with respect to at least one ofthe virtual environment or the virtual object.
 4. The method of claim 2,wherein, in a real environment corresponding to the virtual environment,an object capable of being captured as a real DVS image, which iscapable of being recognized by the image recognition model, is placed inthe virtual environment as the virtual object.
 5. The method of claim 1,wherein the generating of the virtual DVS image comprises: determiningat least one camera view point in the virtual environment generated bythe virtual simulator; and generating at least one virtual DVS imagesimultaneously captured from the at least one camera view point, whereinthe image recognition model is trained based on the at least one virtualDVS image.
 6. The method of claim 1, wherein, a virtual environment isgenerated by the virtual simulator based on information about asurrounding environment in which a real DVS image capable of beingrecognized by the image recognition model is capable of being captured,and the virtual DVS image is generated based on the virtual environment.7. The method of claim 6, wherein, based on the information about thesurrounding environment changing by more than a reference value, thevirtual DVS image is generated based on the changed information aboutthe surrounding environment.
 8. An electronic device configured to trainan image recognition model, the electronic device comprising: a memorystoring the image recognition model; and at least one processorconfigured to: generate a virtual dynamic vision sensor (DVS) imageusing a virtual simulator, generate label information comprisinginformation about a correct answer to a result of recognition of the DVSimage by the image recognition model with respect to the DVS image, andtrain the image recognition model by modifying the image recognitionmodel so that a difference between the result of recognizing the DVSimage by the image recognition model and the label information isreduced.
 9. The electronic device of claim 8, wherein the virtual DVSimage is generated based on a virtual environment generated by thevirtual simulator and a virtual object placed in the virtualenvironment.
 10. 10. The method of claim 9, wherein the labelinformation is obtained based on predefined characteristic informationwith respect to at least one of the virtual environment or the virtualobject.
 11. The electronic device of claim 9, wherein, in a realenvironment corresponding to the virtual environment, an object capableof being captured as a real DVS image, which is capable of beingrecognized by the image recognition model, is placed in the virtualenvironment as the virtual object.
 12. The electronic device of claim 8,wherein the at least one processor is further configured to: determineat least one camera view point in the virtual environment generated bythe virtual simulator and generate at least one virtual DVS imagesimultaneously captured from the at least one camera view point, whereinthe image recognition model is trained based on the at least one virtualDVS image.
 13. The electronic device of claim 8, wherein, a virtualenvironment is generated by the virtual simulator based on informationabout a surrounding environment in which a real DVS image capable ofbeing recognized by the image recognition mode is capable of beingcaptured, and the virtual DVS image is generated based on the virtualenvironment.
 14. The electronic device of claim 13, wherein, based onthe information about the surrounding environment changing by more thana reference value, the virtual DVS image is generated based on thechanged information about the surrounding environment.
 15. Anon-transitory computer-readable recording medium having recordedthereon a program which, when executed by a processor of an electronicdevice, cause the electronic device to perform operations including themethod of claim 1.